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ABSTRACT 



ADAPT [IT] (Advanced Design Approach for Personalized 
Training- Interactive Tools is a European project coordinated by the Dutch 
National Aerospace Laboratory. The aim of ADAPT [IT] is to create and validate 
an effective training design methodology, based on cognitive science and 
leading to the integration of advanced technologies, so that the training 
community can better meet the many challenges of the information society of 
the 21st century. The effort falls within the European Commission's 
Information Society Technologies program. ADAPT [IT] is developing a 
personalized training design methodology, providing associated design tools 
for the efficient realization of that methodology, and validating the 
methodology in different training domains. The 3-year project began in 
February 2000, with an estimated total of 181 person-months required for 
completion. The effort is broken into these 9 work packages: project 
management; design preparation; design of the ADAPT method; development of 
the ADAPT tools; using ADAPT: design process evaluation; using ADAPT: design 
product evaluation; ADAPT [IT] revisions; standardization; and integration, 
dissemination, exploitation. An extensive needs assessment phase has been 
completed. The aim of this effort was to explore current practice in 
instructional design and associated with that, the needs of the training 
designers in order to improve their project and its products. This paper 
reports the needs assessment procedures and outcomes informing how they guide 
the efforts within the project. (Contains 16 references.) (AEF) 
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Abstract: ADAPT*^ is a European project within the Information Society 
Technologies programme that is providing design methods and tools to 
guide training designers according to the latest cognitive science and 
standardization principles. An extensive needs assessment phase has been 
completed. The aim of this effort was to explore current practice in 
instructional design and associated with that, the needs of the training 
designers in order to improve their practice and to insure that real world 
needs will be addressed in the project and its products. This paper reports 
the needs assessment procedures and outcomes informing how they guide 
the efforts within the project. 



Introduction 

rr 

Advanced Design Approach for Personalised Training- Interactive Tools (ADAPT ) is a 
European project coordinated by the Dutch National Aerospace Laboratory (NLR). The aim of 
ADAPT^ is to create and validate an effective training design methodology, based on cognitive 
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science and leading to the integration of advanced technologies, so that the training community 
can better meet the many challenges of the information society of the 21st century. The effort 
falls with the European Commission’s Information Society Technologies programme (1ST; see 
ht^://www.cordis.lu/ist/) and includes as partners: National Aerospace Laboratory, the Open 
University of the Netherlands, the University of Bergen, SevenMoimtains Software, 
EuroControl, the Swedish Air Traffic Control Academy, and Piaggio Aerospace. 

ADAPT^ is developing a personalised training design methodology, providing associated design 
tools for the efficient realisation of that methodology, and validating the methodology in 
different training domains. The three-year project began in February 2000, with an estimated 
total of 181 person-months required for completion. The effort is broken into these 9 work 
packages: 

1 . Project management 

2. Design preparation 

3. Design of the ADAPT method 

4. Development of the ADAPT tools 

5. Using ADAPT : Design process evaluation 

6. Using ADAPT : Design product evaluation 

7. ADAPT^ revisions 

8. Standardisation 

9. Integration, dissemination, exploitation 

The aviation industry is targeted for this research and development effort as it is a key industry 
that is faced with difficult training problems and complexity of skills for which the design 
method is intended. Current challenges for aviation training relate to the increasing complexity 
of dynamic task environments, increasing time constraints, and increasing demands for cognitive 
and information-managing tasks. As technology takes over or automates many basic tasks and 
adds functionalities to operational systems, the result is that more demand is placed on humans to 
perform higher level and supervisory tasks. The problems within aviation training are exemplary 
of highly complex and automated task environments that require flexible skills and are likely to 
forecast training problems in other professional domains. Two quite different training areas 
within the aviation industry (air traffic control and aircraft maintenance) are involved so as to 
insure generalisability of the methodology and the tool to other subject areas and industries (see 
Eseryel & Spector, 2000 for more details). 

Project partners are currently collaborating on designing the ADAPT methodology and 
associated tools (work package 3). The early literature review and training requirements analysis 
indicated that the most appropriate and relevant methodology was the four-component 
instructional design (4C/ID) model (van Merrienboer, 1997). In developing ADAPT 
methodology, the 4C/ID model will be: (1) extended to become a personalised training approach; 
and (2) tailored to meet the needs of actual designers or design teams in a variety of training 
domains, economical sectors, and company sizes. In order to achieve the latter, current practice 
in designing training for complex cognitive skills had to be identified. The project team realised 
this through three different data sources. First, the literature on training designers’ needs was 
reviewed. Then, in order to better understand the instructional design practice of oiu target 



group, questionnaires and extensive interviews were conducted with training designers in the 
aviation industry and in other business and industry sectors across Europe that involved training 
of complex skills (work package 2). In this paper, the findings of this needs assessment phase are 
presented. The next section contains a review of the literature on current practice in instructional 
design together with the findings of the interviews and questionnaires conducted during the 
needs assessment phase. The paper concludes with implications of the needs assessment phase 
for the design and validation of ADAPT methodology and ADAPT^ instructional design tool. 

Current Instructional Design Practice & Needs 

Literature review on current instiuctional design (ID) practice reveals that ID practice is 
significantly different from what is prescribed in ID models (Holcomb, Wedman & Tessmer, 
1996; Klimczak & Wedman, 1997; Loughner & Moller, 1998; Pieters & Bergman, 1995; 
Rowland, 1992; Seels & Glasgow, 1991; Visscher-Voerman, 1999; Wedman & Tessmer, 1993; 
Zemke, 1985). The most prominent activity prescribed in ID models, but rarely completely 
performed by practitioners, is task analysis. Other instructional design activities that are often 
omitted or only partially performed are assessment of entry skills or characteristics, and the pilot 
testing of programs prior to implementation. Studies that investigated expert instructional design 
performance also revealed important differences between the activities expert designers perform 
and what is prescribed in ID models (Maistre, 1998; Perez & Emery, 1995; Rowland, 1992). 

In the literature, several reasons have been identified that might explain the lack of proper 
application of ID models. These reasons can be divided into reasons that are either internal or are 
external to ID (Gros, Elen, Kerres, van Merrienboer & Spector, 1997). Internal reasons refer to 
insufficiencies of current ID models that might be responsible for their limited usefulness. These 
are threefold: (1) the models are not sensitive to the design context; (2) the models do not match 
with the way designers work in practice; and, (3) there are many ID models and they are hard to 
differentiate. On the other hand, external reasons refer to deficiencies in the application of 
existing ID models. These can be attributed to the lack of professionalism in the ID field and lack 
of necessary design skills and knowledge. In order to improve instructional design practice, ID 
tools have been developed to fill this gap (Spector, Poison, & Muraida, 1993). However, what 
have been missing in the area of automated instructional design are sufficiently elaborated design 
models with associated guidance and frameworks that are appropriate for the design of training 
for complex cognitive skills. The ADAPT^ project is intended to fill this gap with the set of 
methods and tools available to instructional designers. For the project to be successful, it is 
important for the design team to know the needs of the probable users of system. Therefore, 
training designer’s need across Europe had to be identified as a next step of needs assessment 
phase in the project. 

Next to the more global literature review, designers’ needs were identified by means of 
questionnaires and interviews (see ADAPT^, 2000). The target group was training designers in 
Europe who are actively involved in the design of training for complex domains. Training 
designer needs are defined as the requirements that have to be met in order to change and 
improve current design practice. These requirements refer primarily to instructional design 
practitioners’ perceived needs for support to use and apply instructional design theories and 
models. 
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The questionnaires were sent to respondents in different sectors: education, information and 
communications technology, transportation, business, and manufacturing. The response rate was 
low (8% of in total 150 questionnaires), which is attributed to the length of the questionnaires 
and the time required to fill them out. Although this may be counted a weakness, the 
questionnaires provided in-depth information to make informed judgments about the target 
group. Besides, the questionnaires were backed up with extensive interviews in order to gather 
more detailed information on current training design practices, the problems that are encountered 
and the desired support and specific needs during the training design process. 

Each interview took around three hours. In total, 10 interviews were conducted involving: 1 
naval college, 3 air traffic control centres, 2 aircraft maintenance organisations, 1 
telecommunications organisation, 1 information technology company, 1 training consultancy 
organisation and 1 distance education company. Unlike the questionnaires that produced data in 
one format, the more in-depth interviews were intended to provide detailed and rich information 
about company design practices. As interviewee background and companies differ substantially, 
the interviews were loosely structured in the sense that questions were open and no pre-defined 
answer categories were supplied. This was intended to stimulate the interviewer towards more 
natural, personal and company-specific responses. Some structure was provided for the 
interviewer by a list of questions and themes to be addressed. The interviews not only yielded 
more detailed information, but, more importantly, they yielded an impression or feeling of 
current design practice, which did not emerge from the questionnaires. 

The emphasis during these data gathering efforts was on the difference between current and ideal 
training design practice, on the problems experienced during the design process, and on the ideal 
characteristics of an instructional design tool. Table 1 provides an overview of the most 
important characteristics of current design practice that emerged from the results of the 
questionnaires and the interviews. The literature review, the questionnaire survey and the 
interviews have also resulted in training designer’s needs for ADAPT*^. The most important 
designer needs for ADAPT*^ are summarised in table 2. 

Current instructional design practice and the training designer’s needs emerged as an outcome of 
needs assessment phase strongly suggests the demand for an integrated ID method-tool 
combination by business and industry. ADAPT*^ project aims at bridging the gap between 
complex training problems and new technological possibilities by developing and validating a 
training design framework to guide the use of state-of-the-art cognitive approaches within 
advanced training systems. In order for the project to be successful, the design effort is guided by 
the outcomes of the needs assessment phase as design requirements (work package 3). These 
outcomes will also be utilized for evaluating the final outcomes of the products (work packages 5 
& 6). The next section discusses the implications of the findings of needs assessment for the 
design and evaluation of ADAPT*^. 
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Category 


Current design practice 


Training designer 
background 


• A majority of training designers have an operational 
background and became designers after having worked in the 
operational field. They are not fully occupied with designing 
training as it is often combined with roles as instructor or 
training manager. 


Training design process 


• Current training design practice does not follow a standard 
Instructional Systems Design model. Activities are carried out 
selectively and often partly and are often characterised by 
implicit and intuitive methods. 

• The analysis phase is not carried out explicitly and in detail, 
mostly due to constraints involving time and resources. 

• The design phase appears often to be combined with the 
development phase, and often happens implicitly as part of 
the development and implementation of training. Specific 
instructional design models or principles are rarely explicitly 
used by designers with an operational background. However, 
many basic ideas and some instructional design principles 
seem to be intuitively understood and used. 

• A detailed and systematic evaluation does not occur and is for 
a large part based on subjective data. The distinction between 
formative and summative evaluation is not found. Formative 
evaluation seems to take place informally. Summative 
evaluation mainly focuses on short-term evaluation. 


Organisational aspects 


• In general, training is not designed in formally structured 
design teams. The most important actors in a team are subject 
matter experts and instructors, who typically design the 
training. Collaborative design does not seem to take place 
explicitly. 



Table 1: Characteristics of current design practice that emerged from the questionnaires and the 
interviews. 



Implications for Design and Evaluation of ADAPT^^ 

rr 

The training designers’ needs that form one of the guiding principles for ADAPT is 
originating from the fact that respondents questioned the merit of a tool as opposed to current 
design practice. Respondents were justifiably concerned with regard to a tool constraining the 
application of their current working methods and knowledge and skills. As instructional 
designers vary along the dimension from novices to experts with corresponding cognitive 
differences in the way they approach their tasks the system should be flexible enough (i.e., weak 
approach to design) to handle different sequence of uses and the messages should be tailored to 
the designer’s level of expertise. ADAPT^ should therefore be set up in such a way that all 
designers, including designers without a specific educational background, should quickly 
become aware of and convinced of the additional value of ADAPT^. The system should be 
flexible enough to allow advanced users integrate their own design practise, knowledge and 
skills. Inexperienced designers may require (ask for) more support in the process of structuring 
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Category 


Training Designer's Needs 


Training designer 

background 


1. Target at training designers with an operational background 
but do not exclude those designers with an educational 
background; 

2. Use non-academic instructional design language, fitting the 
main target group of designers with an operational 
background. 


Training design process 


1 . Support an explicit, structured and systematic design process, 
especially for the analysis, design and evaluation phases; 

2. Take into account that current design practice is often 
characterised by implicit and intuitive methods. Support the 
designers in the change in design process (e.g. by allowing 
some intermediate ways of design focusing on the vital aspects 
of training design); 

3. Convince the users to adapt their working practice (e.g. by 
explaining the added value of the different design steps and 
explaining the steps by means of practical worked-out 
examples); 

4. Provide pragmatic, practical useful methods that yield quick 
results and can be easily used to produce at least a minimal 
required training design (see also 2). 


Organisational aspects 


1. Address the relational aspects during the training design 
process (setting up teams, involving users, methods for 
teamwork, collaborative design, communication etc.). 


Design tool characteristics 


1 . Provide practical worked-out examples; 

2. Provide support/guidance in applying instructional design 
principles; 

3 . Link together the different training design products; 

4. Provide methods for file management and version control; 

5 . Make information easily re-usable and retrievable; 

6. Allow different degrees of freedom (e.g. from structured to 
non-sequential design); 

7. Increased efficiency: design training more quickly; 

8. Increased effectiveness: design better training. 



Table 2: Some of the training designer ’s needs identified by the questionnaire survey and the 
interviews. 



their analysis and design results. For this kind of audience, the use of non-academic language is 
of utmost importance. In addition, a number of design supports will be integrated in the system: 

1. The ADAPT methodology handbook that will describe how to analyse complex 
cognitive skills and how to design training for those skill along with worked-out 
examples. 

2. The CORE software tool that will support the use of the ADAPT methodology by 
allowing storing and viewing analysis and design results and by producing a blueprint 
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for training in such format that it can be further worked out with third party authoring 
software. 

3. An online help system and manuals, describing functionality and procedures for using 
the ADAPT tool with implementation examples. 

4. An expert design and standards advisor. 

5. An evaluation software (EVAL) to support formative and summative evaluation of 
the training. 

6. Familiarisation training for the trainers to teach to apply the ADAPT method and tool 
supported by a web-based online tutorial. 

The ADAPT system should have an impact on training design practice. Therefore, the method- 
tool combination is aimed at ensuring effective training design for complex cognitive skill 
embodying a validated training design methodology for personalized training that is based on the 
latest developments in the cognitive science. The tool itself also aims at more efficient training 
design by integrating several technologies, such as the re-usability of intermediate design 
products, version control, and support for adapting training design to changes in the external 
constraints (e.g., training time, characteristics of the target trainee, etc.). Furthermore, the system 
will be customisable to user preferences and allow collaborative design and rapid prototyping. 

The project recognizes evaluation for its critical role it plays throughout the lifetime of the 
system. Functional specifications arising from the needs assessment phase and the aims of the 
system forms the basis of the evaluation plan guiding the design, production and revision of the 
full-scale system. The evaluation and validation activities in the project are twofold: (1) 
evaluation of the training design process; (2) validation of effectiveness of training design 
products. Integrated in the former, the instructional design modelling approach will also provide 
the means to pinpoint problems in instructional design guidance that the system provides. In both 
studies, the impact of the ADAPT^ system will be compared with the traditional practices. The 
project partners from aircraft maintenance and air traffic control will be involved in these studies 
to insure the generalisability of the methods and the tools to other domains. The user-centred 
process adopted for this project insures that the real world needs will be addressed in the project 
and its products. 
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